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In this tutorial we schematically illustrate four algorithms: 

1. ABC rejection for parameter estimation [1, 2], 

2. ABC SMC for parameter estimation [3, 4, 5], 

3. ABC rejection for model selection on the joint space [6], 

4. ABC SMC for model selection on the joint space [7]. 

We suggest to read this tutorial from the beginning. We start with a detailed 
explanation of the ABC rejection algorithm, which later helps to understand ABC SMC 
as it is based on the same concepts. Also, both model selection algorithms are closely 
related to parameter estimation algorithms and it is therefore helpful to understand 
those first. 

This tutorial forms a part of the supplementary material of the paper "Simulation- 
based model selection for dynamical systems in systems and population biology, 
Bioinformatics, 26 (1), 104-110, 2010" (T. Toni, M. P. H. Stumpf). 
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ABC rejection 



(a) We define a prior distribution P(0) 
and we would like to approximate the posterior 
distribution P(0\Dq). We start by sampling a 
parameter #* from the prior distribution. We 
call this sampled parameter a particle. 

(b) We simulate a data set D* according 
to some simulation framework f(D\0*). In 
our examples we use different simulation 
frameworks. If we simulate a deterministic 
dynamical model, we add some noise at the 
time points of interest. If we simulate a 
stochastic dynamical model, we do not add 
any additional noise to the trajectories. We 
compare the simulated data set D* (circles) 
to the experimental data Do (crosses) using 
a distance function, <i, and tolerance e; if 
d(D ,D*) < e, we accept 0*. The tolerance 
e > is the desired level of agreement between 
D and D*. 

(c) The particle #* is accepted because 
D* and Do are sufficiently close. 

(d) We sample another parameter #* from the 
prior distribution and simulate a corresponding 
dataset D*. In this case D* and Dq are very 
different and we reject the particle (we "throw 
it away"). 

(e) We repeat the whole procedure until 
N particles have been accepted. They repre- 
sent a sample from P(9\d(Do, D*) < e), which 
approximates the posterior distribution. If 
e is sufficiently small then the distribution 
P(6\d(D ,D*) < e) will be a good approxi- 
mation for the "true" posterior distribution, 

p(e\D ). 

(f) Many particles were rejected in the 
procedure, for which we have spent a lot of 
computational effort for simulation. ABC 
rejection is therefore computationally ineffi- 
cient. We can use ABC SMC to reduce the 
computational cost. 
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Figure 1: Schematic representation of ABC re- 
jection. 
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ABC SMC (Toni et a/., 2009) 

(a) As in ABC rejection, we define a prior 
distribution P(6) and we would like to approxi- 
mate a posterior distribution P(0\Dq). In ABC 
SMC we do this sequentially by constructing 
intermediate distributions, which converge 
to the posterior distribution. We define a 
tolerance schedule e\ > 62 > . . . ct > 0. 
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(b) We sample particles from a prior distribu- 
tion until N particles have been accepted (have 
reached the distance smaller than ei). For all 
accepted particles we calculate weights (see 
[4] for formulas and derivation). We call the 
sample of all accepted particles "Population 
1". 
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(c) We then sample a particle 0* from popu- 
lation 1 and perturb it to obtain a perturbed 
particle #** ~ K(0\0*), where K is a per- 
turbation kernel (for example a Gaussian 
random walk). We then simulate a dataset 
D* ~ f(D\9**) and accept the particle (9** 
if d(D ,D**) < e 2 . We repeat this until we 
have accepted TV particles in population 2. We 
calculate weights for all accepted particles. 
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(d) We repeat the same procedure for the 
following populations, until we have accepted 
N particles of the last population T and 
calculated their weights. Population T is a 
sample of particles that approximates the 
posterior distribution. 

ABC SMC is computationally much more 
efficient than ABC rejection (see [4] for 
comparison). 
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Figure 2: Schematic representation of ABC 
SMC. 
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ABC rejection for model selection 
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A sample from the prior distribution (white) and accepted particles (blue) 
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Approximation of P(mlDo) 
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Figure 3: (a) Prior and posterior distributions, P(ra, 0) and P(m,6\Do), are now defined on a 
joint model and parameter space, (b) Particles (m, 6) are sampled from the prior distribution and 
accepted/rejected according to the distance between the simulated and experimental datasets. 
The accepted particles are shown in dark blue, (c) Six particles have been accepted: one from 
model 1 and five from model 2. The approximated marginal posterior probability of the model 
can be calculated as P(m = 1\Dq) = |, P(m = 2\Dq) = §? P{m = 3\Dq) = | . For illustrative 
purposes we have chosen a small number of particles. In principle this algorithm will yield 
consistent marginal posterior model distributions for N — »• oo. 
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ABC SMC for model selection 



A sample from the prior distribution and accepted particles of Popl 
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Population t-1 
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Accept/reject (m**, 6**). 

If accepted, calculate the weights. 
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Figure 4: 



(a) Particles are sampled from the prior P(m, 0) until N particles have been accepted 
(in this example TV = 6 for illustration purposes, in practice N should be larger), that is the 
distance is smaller than e\. The weights w t =i are calculated for all accepted particles and 
normalized. 

(b) To obtain the marginal probability of m, we sum over all the weights corresponding 
to the model of interest: P £=1 (m / ) = J2m=m' ^£=i( m ? The histogram of the marginal 
population 1 is presented. 

(c) This figure shows how we propose and accept a particle of population t, t = 2, . . . , T. We 
sample a model m* from population t — 1 with probability P £ _i(m*). For example, we might 
have sampled m* = m2. We then perturb the model using a model perturbation kernel KM t 
to obtain m** ~ i^M £ (m|m*), for example m** = ms. After we have obtained the model 
m**, we sample a parameter 0* belonging to model m** from population t — 1 and perturb 
it to obtain 6** ~ KP t}7n ** (0\0*). We simulate a dataset D* for a particle (m**,6>**) and 
accept (d(Do, D*) < e t ) or reject the particle. If a particle is accepted, we calculate its weight 
w t (m**,0**). 

(d) When N particles of population t have been accepted, we normalize the weights and 
marginalize them in order to obtain the marginal intermediate population of the model P t (m). 
We continue until population T, which is the approximation of the joint posterior distribution 
P(m,9\Do). The quantities of interest are the intermediate and the last marginal population of 
the model. In this example there are three populations, T = 3. 
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